Skip to content

fix(repair): stop dropping drawers with zero embedding_metadata rows - #2087

Open
KeilerHirsch wants to merge 1 commit into
MemPalace:developfrom
KeilerHirsch:fix/repair-sqlite-bypass-sparse-metadata
Open

fix(repair): stop dropping drawers with zero embedding_metadata rows#2087
KeilerHirsch wants to merge 1 commit into
MemPalace:developfrom
KeilerHirsch:fix/repair-sqlite-bypass-sparse-metadata

Conversation

@KeilerHirsch

Copy link
Copy Markdown
Contributor

Problem

extract_via_sqlite (the recovery path rebuild_from_sqlite uses when the chromadb collection layer itself is broken, e.g. the #1308 HNSW-segment failure) drives its extraction with an INNER JOIN that starts at embedding_metadata:

FROM embedding_metadata em
JOIN embeddings e ON em.id = e.id
WHERE e.segment_id = ?

Any embedding with zero rows in embedding_metadata — a sparse historical write with no chroma:document key and no other key — never appears in the join result at all, so it is silently dropped.

This condition is not hypothetical: the sibling extractor _extract_drawers (used by the collection-layer rebuild_index path) already sanitizes for it directly in its own comments — "drawers extracted from sqlite ground truth can come back with None or {} for sparse historical writes" (#1458). extract_via_sqlite has no equivalent handling, because such rows are structurally invisible to a join anchored on embedding_metadata.

Running mempalace repair --mode from-sqlite --archive-existing on a palace containing any such drawer drops it with no count mismatch and no warning.

Fix

Drive the query from embeddings (LEFT JOIN embedding_metadata) instead, defaulting to an empty metadata dict when a row has none. Order by e.id rather than em.id, since em.id is NULL for the now-included unmatched rows.

Tests

One new regression test in tests/test_repair.py: seeds a real chromadb collection via the existing _seed_palace helper, then strips one drawer's embedding_metadata rows directly via SQLite (current chromadb validates against empty metadata on write, so this only reproduces on rows already sitting in an older palace) and asserts the drawer still comes back through the SQLite bypass. Confirmed failing against the pre-fix code, passing after the fix. All 97 existing tests in tests/test_repair.py still pass.

extract_via_sqlite drove its extraction with an INNER JOIN starting at
embedding_metadata. Any embedding with zero rows there (a sparse
historical write with no chroma:document and no other key -- the same
condition _extract_drawers already sanitizes for the collection-layer
rebuild path, see MemPalace#1458) never appeared in the join result and was
silently dropped by mempalace repair --mode from-sqlite, with no count
mismatch and no warning.

Drive the query from embeddings (LEFT JOIN embedding_metadata) instead,
defaulting to an empty metadata dict when a row has none. Order by
e.id rather than em.id since em.id is NULL for unmatched rows.

Regression test seeds a real chromadb collection, then strips one
drawer's metadata rows directly via SQLite (current chromadb validates
against empty metadata on write, so this only reproduces on rows
already sitting in an older palace) and asserts the drawer still
comes back through the SQLite bypass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant